Statistical and Machine Learning Models to Predict Programming Performance

This thesis details a longitudinal study on factors that influence introductory programming success and on the development of machine learning models to predict incoming student performance. Although numerous studies have developed models to predict programming success, the models struggled to achieve high accuracy in predicting the likely performance of incoming students. Our approach overcomes this by providing a machine learning technique, using a set of three significant factors, that can predict whether students will be ‘weak’ or ‘strong’ programmers with approximately 80% accuracy after only three weeks of programming experience. This thesis makes three fundamental contributions. The first contribution is a longitudinal study identifying factors that influence introductory programming success, investigating 25 factors at four different institutions. Evidence of the importance of mathematics, comfort-level and computer game-playing as predictors of programming performance is provided. A number of new instruments were developed by the author and a programming self-esteem measure was shown to out-perform other previous comparable comfort-level measures in predicting programming performance. The second contribution of the thesis is an analysis of the use of machine learning (ML) algorithms to predict performance and is a first attempt to investigate the effectiveness of a variety of ML algorithms to predict introductory programming performance. The ML models built as part of this research are the most effective models so far developed. The models are effective even when students have just commenced a programming module. Consequently, timely interventions can be put in place to prevent struggling students from failing. The third contribution of the thesis is the recommendation of an algorithm, based on detailed statistical analysis that should be used by the computer science education community to predict the likely performance of incoming students. Optimisations were carried out to investigate if prediction accuracy could be further increased and an ensemble algorithm, StackingC, was shown to improve prediction performance. The factors identified in this thesis and the associated machine learning models provide a means to predict accurately programming performance when students have only completed preliminary programming concepts. This has not previously been possible

CS1: how will they do? How can we help? A decade of research and practice

Background and Context: Computer Science attrition rates (in the western world) are very concerning, with a large number of students failing to progress each year. It is well acknowledged that a significant factor of this attrition, is the students’ difficulty to master the introductory programming module, often referred to as CS1. Objective: The objective of this article is to describe the evolution of a prediction model named PreSS (Predict Student Success) over a 13-year period (2005–2018). Method: This article ties together, the PreSS prediction model; pilot studies; a longitudinal, multi-institutional re-validation and replication study; improvements to the model since its inception; and interventions to reduce attrition rates. Findings: The outcome of this body of work is an end-to-end real-time web-based tool (PreSS#), which can predict student success early in an introductory programming module (CS1), with an accuracy of 71%. This tool is enhanced with interventions that were developed in conjunction with PreSS#, which improved student performance in CS1. Implications: This work contributes significantly to the computer science education (CSEd) community and the ITiCSE 2015 working group’s call (in particular the second grand challenge), by re-validating and developing further the original PreSS model, 13 years after it was developed, on a modern, disparate, multi-institutional data set

An Analysis of Alternative Approaches for the Distribution of Lecture Notes with the Aid of a Virtual Learning Environment to Promote Class Engagement

The use of Virtual Learning Environments (VLEs) has become popular over the last ten years at third level institutions. At NUIM the Moodle VLE is used to disseminate lecture notes, share course related resources, perform assessment, and provide a means for online communication. This paper is interested in how effectively to use a VLE to disseminate lecture notes. At NUIM lectures notes are typically posted on Moodle before a lecture (for example, all notes posted at the start of the semester or several days before an upcoming lecture etc.) or after a lecture with lecturers having a personal preference for a particular method. In this paper a pilot study on the dissemination of lectures notes through Moodle to a large first year undergraduate class is described. In previous years student disengagement in this class has been an issue, therefore, three different approaches were trialled in this pilot. In the first approach, a summary set of lecture notes to be covered at the next lecture were made available four days in advance. The summary was a one-page document containing at most six slides of the most important material. The students were encouraged to read the notes in advance. It was hoped that this would lead to more active participation by the students as they had time to assimilate the material prior to the lecture. In the second approach the students were informed in advance that an in-class assessment would be carried out based on the summary. In both instances a full set of notes were made available on Moodle after the lecture. An overview of the findings of this pilot is presented, including data on student participation during both approaches. In addition, a critique of the potential effects on student results is provided and recommendations based on the findings are discussed

Using Support Vector Machines and Acoustic Noise Signal for Degradation Analysis of Rotating Machinery

An automated approach to degradation analysis is proposed that uses a rotating machine’s acoustic signal to determine Remaining Useful Life (RUL). High resolution spectral features are extracted from the acoustic data collected over the entire lifetime of the machine. A novel approach to the computation of Mutual Information based Feature Subset Selection is applied, to remove redundant and irrelevant features, that does not require class label boundaries of the dataset or spectral locations of developing defect to be known or pre-estimated. Using subsets of the feature space, multi-class linear and Radial Basis Function (RBF) Support Vector Machine (SVM) classifiers are developed and a comparison of their performance is provided. Performance of all classifiers is found to be very high, 85 to 98%, with RBF SVMs outperforming linear SVMs when a smaller number of features are used. As larger numbers of features are used for classification, the problem space becomes more linearly separable and the linear SVMs are shown to have comparable performance. A detailed analysis of the misclassifications is provided and an approach to better understand and interpret costly misclassifications is discussed. While defining class label boundaries using an automated k-means clustering algorithm improves performance with an accuracy of approximately 99%, further analysis shows that in 88% of all misclassifications the actual class of failure had the next highest probability of occurring. Thus, a system that incorporates probability distributions as a measure of confidence for the predicted RUL would provide additional valuable information for scheduling preventative maintenance

A neurofeedback system to promote learner engagement

This report describes a series of experiments that track novice programmer's engagement during two attention based tasks. The tasks required participants to watch a tutorial video on introductory programming and to attend to a simple maze game whilst wearing an electroencephalogram (EEG)device called the Emotiv EPOC. The EPOC's proprietary software includes a system which tracks emotional state (specifically: engagement, excitement, meditation, frustration, valence and long-term excitement). Using this data, a software application written in the Processing language was developed to track user's engagement levels and implement a neurofeedback based intervention when engagement fell below an acceptable level. The aim of the intervention was to prompt learners who disengaged with the task to re-engage. The intervention used during the video tutorial was to pause the video if a participant disengaged significantly. However other interventions such as slowing the video down, playing a noise or darkening/brightening the screen could also be used. For the maze game, the caterpillar moving through the maze slowed in line with disengagement and moved more quickly once the learner re-engaged. The approach worked very well and successfully re-engaged participants, although a number of improvements could be made. A number of interesting findings on the comparative engagement levels of different groups e.g. by gender and by age etc. were identified and provide useful pointers for future research studies

The influence of motivation and comfort-level on learning to program

This paper documents a study, carried out in the academic year 2004-2005, on the role of motivation and comfort-level in a first year object-oriented programming module. The study found that intrinsic motivation had a strong correlation with programming performance as did self-efficacy for learning and performance, r=0.512, p < 0.01 and r=0.567, p < 0.01 respectively. Aspects of comfort level were found to have significant correlations with performance with an instrument on programming-esteem rendering the most interesting results. A regression model based upon these factors was able to account for 60% of the variance in programming performance results

A study on alternative strategies for sharing lecture notes using a VLE to promote in-class participation

The use of Virtual Learning Environments (VLEs) has become popular over the last ten years at third level institutions. At the National University of Ireland Maynooth (NUIM) the Moodle VLE is used to disseminate lecture notes, share course related resources, perform assessment, and provide a means for online communication. This paper is interested in how to effectively use a VLE to disseminate lecture notes. At NUIM lectures notes are typically posted on Moodle before a lecture (for example, all notes posted at the start of the semester or several days before an upcoming lecture etc.) or after a lecture with lecturers having a personal preference for a particular method. In this paper a pilot study on the dissemination of lectures notes through Moodle to a large first year undergraduate class is described in this paper. In previous years student disengagement in this class has been an issue. As such two different approaches were trialed. In the first approach, a summary set of lecture notes to be covered at the next lecture were made available four days in advance. The summary was a one-page document containing at most six slides of the most important material in the lecture. The students were encouraged to read the notes in advance. It was hoped that this would lead to more active participation by the students as they had time to assimilate the material prior to the lecture. In the second approach the students were informed in advance that an in-class assessment would be carried out based on the summary. In both instances a full set of notes were made available on Moodle after the lecture. An overview of the findings of this pilot is presented, including data on student participation during both approaches. In addition, a critique of the potential effects on student results is provided and recommendations based on the findings are discussed

The influence of motivation and comfort-level on learning to program

This paper documents a study, carried out in the academic year 2004-2005, on the role of motivation and comfort-level in a first year object-oriented programming module. The study found that intrinsic motivation had a strong correlation with programming performance as did self-efficacy for learning and performance, r=0.512, p < 0.01 and r=0.567, p < 0.01 respectively. Aspects of comfort level were found to have significant correlations with performance with an instrument on programming-esteem rendering the most interesting results. A regression model based upon these factors was able to account for 60% of the variance in programming performance results

Using an innovative assessment approach on a real-world group based software project

Currently, there is a lack of practical, real-world projects on Computer Science (CS) courses at Maynooth University. Generally CS undergraduate modules are composed of 24 hours of lectures and 24 hours of labs where students learn theoretical concepts in the lectures and apply their understanding to practical lab-based exercises. The problem with this approach is that students do not gain any awareness of, or learn how to solve tasks that they are likely to encounter in a real-world industrial setting; nor do they gain experience of working as part of a team even though most software development positions involve team-based work. This paper reports on a web-based development module that incorporated a real-world group based project was re-designed and delivered. The module went well; however, assessing the work fairly was found to be difficult, especially where team members contributed at considerably varying levels was a challenge. Of particular concern was that some hard-working students were penalised by other students poor work and lazy students were rewarded because of more hard-working students work. This action research project will attempt to re-address how to assess this group-based work with a cohort of students. The goal of the research is to implement an innovative assessment structure, using peer-, self-, and co-assessment, for a group based real-world project, that is deemed fair and reasonable and provided a good learning environment

On the benefits of philosophy as a way of life in a general introductory course

Philosophy as a way of life (PWOL) places investigations of value, meaning, and the good life at the center of philosophical investigation, especially of one’s own life. I argue PWOL is compatible with general introductory philosophy courses, further arguing that PWOL-based general introductions have several philosophical and pedagogical benefits. These include the ease with which high impact practices, situated skill development, and students’ ability to ‘think like a disciplinarian’ may be incorporated into such courses, relative to more traditional introductory courses, as well as the demonstration of philosophy’s value to students by explicitly tying philosophical investigation to students own lives